Visualizing Offensive language keywords - Word2Vec and t-SNE

Slavko Zitnik, March 4, 2021

Code is adapted from Google News and Leo Tolstoy: Visualizing Word2Vec Word Embeddings using t-SNE by Sergey Smetanin.

We use Google's Word2Vec vectors from https://code.google.com/archive/p/word2vec/.

We are publishing pre-trained vectors trained on part of Google News dataset (about 100 billion words). The model contains 300-dimensional vectors for 3 million words and phrases. The phrases were obtained using a simple data-driven approach described by Mikolov et al., 2013.

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed Representations of Words and Phrases and their Compositionality. In Proceedings of NIPS, 2013.

In [1]:
import gensim
import gensim.downloader
from sklearn.manifold import TSNE
import numpy as np
import tempfile
import imageio
import shutil
import os
from statistics import mean
import pandas as pd 

from IPython.display import Image
from IPython.display import display
pd.options.display.max_columns = None

import matplotlib.patheffects as PathEffects
import matplotlib.pyplot as plt
import matplotlib.cm as cm
%matplotlib inline

from matplotlib.axes._axes import _log as matplotlib_axes_logger
matplotlib_axes_logger.setLevel('ERROR')
In [2]:
model_gn = gensim.downloader.load('word2vec-google-news-300')
In [3]:
keys = [ \
    "profane",
    "slur",
    "derailing",
    "harrasment",
    "stereotype",
    "spam",
    "obscene",
    "dominance",
    "threat",
    "discredit",
    "hateful", # hate-speech not in dictionary, should be taken into account together
    "insult",
    "benevolent",
    "hostile"
]
In [4]:
embedding_clusters = []
word_clusters = []
for word in keys:
    embeddings = []
    words = []
    for similar_word, _ in model_gn.most_similar(word, topn=30):
        words.append(similar_word)
        embeddings.append(model_gn[similar_word])
    embedding_clusters.append(embeddings)
    word_clusters.append(words)

Top 30 neighbouring words based on key words.

In [5]:
df = pd.DataFrame(dict(zip(keys, word_clusters)))  
display(df)
profane slur derailing harrasment stereotype spam obscene dominance threat discredit hateful insult benevolent hostile
0 vulgar racial_slur derail harrassment stereotypes spamming vulgar domination threats discrediting bigoted insulting beneficent unfriendly
1 profanity racist_slur derailed harassment stereotypical spammers indecent supremacy danger malign racist affront benevolence Hostile
2 obscenities slurs sabotaging Harassment stereotyped antispam Obscene stranglehold threat_posed besmirch vile insulted magnanimous antagonistic
3 vulgar_language derogatory stalling harassments negative_stereotypes Spam pornographic superiority imminent_danger embarrass hurtful disrespect beneficence nipple_pinching
4 foul_language racist_remark thwarting sexual_harrasment stereotyping spam_filters lewd dominant Threat delegitimize mean_spirited Insult altruistic warlike
5 expletives epithet scuppering harrasing gender_stereotypes spam_mails sexually_explicit hegemony existential_threat demonize vile_hateful disrespectful benevolently unwelcoming
6 curse_words derogatory_remark undermining harassement stereotypically phishing filthy_diatribe dominate menace smear anti_semetic disgrace saintly hostility
7 hateful racial_slurs wrecking sexual_harrassment sterotype malware vulgar_obscene ascendancy pose_threat marginalize bigotry denigrate malevolent mistaken_celebratory_gunfire
8 foul_mouthed word_nigger disrupting ASPEN_Colo._Actor preconceptions spam_filtering obscenity pre_eminence graver_threat vilify vitriolic demean benevolent_ruler confrontational
9 profanities remark scuttling Harrassment myth spim inappropriate preeminence gravest_threat denigrate racist_sexist_homophobic humiliate generous inhospitable
10 politically_incorrect racial_epithet spoiling Verbal_abuse preconceived_notions spammer disgusting dominace challenge belittle hate_mongering gratuitous_insult munificent belligerent
11 containing_advertising_astroturfing derogatory_language scuttle violance sterotypes spyware outrageous dominating Threats defame hatred dishonor tyrannical nonhostile
12 scatological_references racist_connotations ruining sexual_harassments sexist_stereotypes Spim insulting dominancy dangers intimidate anti_Semitic disrespecting bighearted nonhostile_causes
13 Ad_Age_reserves racist jeopardizing verbal_abuse notion spams innapropriate dominated hazard mislead racist_bigoted insults noble neocolonialist_enemies
14 vulgarity homophobic_slurs halting harrassing notions Pushdo_botnet objectionable ascendency nightmare_scenario impugn homophobic belittle virtuous donned_riot_gear
15 sarcastic homophobic_slur stymie harassing racial_stereotypes SPAM sexually_suggestive Dominance possibility undermine bigots taunt tutelary unsupportive
16 swearwords disparaging derails Mr._Lutfi_drugged Stereotypes Spam_filtering vile predominance peril smear_campaign hateful_rhetoric belittling omniscient_omnipotent hostile_takeover
17 vulgar_obscene racist_epithet spurring Rathore_behest prejudices sender_authentication demeaning prowess concern refute bigot_homophobe disparage avaricious bellicose
18 rude racially_derogatory sabotage homophobic_taunts racial_stereotype spam_phishing sexually_exploitive supremecy threatened debunk hatefully disrespects enlightened retaliatory
19 cusswords derogatory_remarks toppling slander perception blocklists pornography duopoly dangers_posed insinuate racists denigrating charitable mutated_creatures
20 mildly_suggestive sexist_slurs scuttled harassment_intimidation archetype antispam_filters immoral monopoly Sadequee_countered humiliate sexist_homophobic Adding_insult omnipotent thinly_veiled_barb
21 scatological pejorative impeding brickbatting dumb_hick Pushdo offensive_hateful strangehold cyberthreat tarnish religiously_intolerant use_racial_slurs selfless adversarial
22 epithets fagot averting forcible_conversion preconceived_notion Spam_filters disseminating_pornographic primacy risk cast_aspersions vitriol_spewed slur munificence hospitable
23 Profanity disparaging_remarks torpedoing anti_semitic_tirade myths_perpetuated spoofing_phishing sexually_exploitative resurgent threaten delegitimise racist_sexist offend pious intimidating
24 vitriolic sexist_slur destroying discrimation cliché CAN_SPAM Libelous mastery undeterrable divert_attention hatefulness shame benevolent_dictator violent
25 profane_vulgar nigger snuffing non_cognisable_offense preconception Srizbi_botnet inapropriate ascendance imminent rebut sexist_racist_homophobic travesty righteous intimidatory
26 profane_abusive racist_slurs delaying Carlos_Irwin_Estevez pre_conceived_notions spammed vulgarity iron_grip specter implicate meanspirited demeaning paternalistic friendly
27 Avoid_lewd_obscene homosexual_slur dooming brutalities_meted stereotypical_portrayals spam_viruses_worms reprehensible dominates mortal_danger subvert bigotted absolute_disgrace tenderhearted conciliatory
28 Foul_language tirade marring harrased typecasts botnets Lewd quasi_monopoly scare misinform hate unkind eleemosynary uncongenial
29 No_vulgarity_racial racial_connotations deliberately_sabotaging Eve_teasing misconception malicious_code obsenity utter_domination problem disinformation racist_bigot offends noblesse_oblige racially_hostile
In [6]:
embedding_clusters = np.array(embedding_clusters)
n, m, k = embedding_clusters.shape
tsne_model_en_2d = TSNE(perplexity=15, n_components=2, init='pca', n_iter=3500, random_state=32)
embeddings_en_2d = np.array(tsne_model_en_2d.fit_transform(embedding_clusters.reshape(n * m, k))).reshape(n, m, 2)
In [7]:
def tsne_plot_similar_words(title, labels, embedding_clusters, word_clusters, a, filename=None):
    plt.figure(figsize=(16, 9))
    colors = cm.rainbow(np.linspace(0, 1, len(labels)))
    for label, embeddings, words, color in zip(labels, embedding_clusters, word_clusters, colors):
        x = embeddings[:, 0]
        y = embeddings[:, 1]
        plt.scatter(x, y, c=color, alpha=a, label=label)
        for i, word in enumerate(words):              
            plt.annotate(word, alpha=0.5, xy=(x[i], y[i]), xytext=(5, 2),
                         textcoords='offset points', ha='right', va='bottom', size=8)
        plt.annotate(label.upper(), alpha=1.0, xy=(mean(x), mean(y)), xytext=(0, 0),
            textcoords='offset points', ha='center', va='center', size=15)
    plt.legend(loc=4)
    plt.title(title)
    plt.grid(False)
    if filename:
        plt.savefig(filename, format='png', dpi=150, bbox_inches='tight')
    plt.show()


tsne_plot_similar_words('Similar words from Google News', keys, embeddings_en_2d, word_clusters, 0.7,
                        'similar_words.png')

Visualizing Word2Vec Vectors

In [8]:
words = keys
embeddings = []
for word in words:
    embeddings.append(model_gn[word])
    
tsne_2d = TSNE(perplexity=30, n_components=2, init='pca', n_iter=3500, random_state=32)
embeddings_2d = tsne_2d.fit_transform(embeddings)
In [9]:
def tsne_plot_2d(label, embeddings, words=[], a=1):
    plt.figure(figsize=(16, 9))
    colors = cm.rainbow(np.linspace(0, 1, 1))
    x = embeddings[:,0]
    y = embeddings[:,1]
    plt.scatter(x, y, c=colors, alpha=a, label=label)
    for i, word in enumerate(words):
        plt.annotate(word, alpha=0.3, xy=(x[i], y[i]), xytext=(5, 2), 
                     textcoords='offset points', ha='right', va='bottom', size=10)
    plt.legend(loc=4)
    plt.grid(True)
    plt.savefig("hhh.png", format='png', dpi=150, bbox_inches='tight')
    plt.show()

tsne_plot_2d('Keywords 2D embeddings visualization', embeddings_2d, words, a=1)

Animation

We take top 200 closest neighours and visualize multiple t-SNE visualizations.

In [10]:
keys_gif = keys

embedding_clusters_gif = []
word_clusters_gif = []
for word in keys:
    embeddings = []
    words = []
    for similar_word, _ in model_gn.most_similar(word, topn=200):
        words.append(similar_word)
        embeddings.append(model_gn[similar_word])
    embedding_clusters_gif.append(embeddings)
    word_clusters_gif.append(words)
In [11]:
embedding_clusters_gif = np.array(embedding_clusters_gif)
n, m, k = embedding_clusters_gif.shape
In [12]:
def tsne_plot_similar_words_png(title, embedding_clusters, keys, a, filename):
    plt.figure(figsize=(16, 9))
    colors = cm.rainbow(np.linspace(0, 1, len(embedding_clusters)))
    
    for embeddings, color, key in zip(embedding_clusters, colors, keys):
        x = embeddings[:, 0]
        y = embeddings[:, 1]
        plt.scatter(x, y, c=color, alpha=a)
        plt.text(x.mean(), y.mean(), key.upper(), color='white', weight='bold', fontsize=13, path_effects=[PathEffects.withStroke(linewidth=3,
                                                                          foreground="black", alpha=0.7)])
    plt.title(title)
    plt.grid(True)
    plt.xlim(-200, 200)
    plt.ylim(-200, 200)
    plt.savefig(filename, format='png', dpi=150, bbox_inches='tight')
    

dirpath = tempfile.mkdtemp()
images = []
for i in range(1, 30):
    fname = os.path.join(dirpath, str(i) + '.png')
    tsne_model_en_2d_gif = TSNE(perplexity=i, n_components=2, init='pca', n_iter=3500, random_state=32)
    embeddings_en_2d_gif = np.array(tsne_model_en_2d_gif.fit_transform(embedding_clusters_gif.reshape(n * m, k))).reshape(n, m, 2)
    tsne_plot_similar_words_png('Vizualizing similar words from Google News using t-SNE (perplexity={})'.format(i), embeddings_en_2d_gif, keys_gif, 0.6, fname)
    images.append(imageio.imread(fname))
    print(f"Finished iteration {i}.")
imageio.mimsave("2d1.gif", images, duration = 0.5)
shutil.rmtree(dirpath)

Finished iteration 1.
Finished iteration 2.
Finished iteration 3.
Finished iteration 4.
Finished iteration 5.
Finished iteration 6.
Finished iteration 7.
Finished iteration 8.
Finished iteration 9.
Finished iteration 10.
Finished iteration 11.
Finished iteration 12.
Finished iteration 13.
Finished iteration 14.
Finished iteration 15.
Finished iteration 16.
Finished iteration 17.
Finished iteration 18.
Finished iteration 19.
Finished iteration 20.

<ipython-input-12-2072e6306bba>:2: RuntimeWarning: More than 20 figures have been opened. Figures created through the pyplot interface (`matplotlib.pyplot.figure`) are retained until explicitly closed and may consume too much memory. (To control this warning, see the rcParam `figure.max_open_warning`).
  plt.figure(figsize=(16, 9))

Finished iteration 21.
Finished iteration 22.
Finished iteration 23.
Finished iteration 24.
Finished iteration 25.
Finished iteration 26.
Finished iteration 27.
Finished iteration 28.
Finished iteration 29.
In [13]:
Image(filename="2d1.gif")
Out[13]:
<IPython.core.display.Image object>